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Abstract 

This paper reports on workload and situation 
awareness of pilots and controllers participating in a 
human-in-the-loop simulation using three different 
distributed air-ground traffic management concepts. 
Eight experimental pilots started the scenario in an 
en-route phase of flight and were asked to avoid 
convective weather while performing spacing and 
merging tasks along with a continuous descent 
approach (CDA) into Louisville Standiford Airport 
(SDF). Two controllers managed the sectors through 
which the pilots flew, with one managing a sector 
that included the Top of Descent, and the other 
managing a sector that included the merge point for 
arrival into SDF. At 3-minute intervals in the 
scenario, pilots and controllers were probed on their 
workload or situation awareness. We employed one 
of three concepts of operation that distributed 
separation responsibility across human controllers, 
pilots, and automation to measure changes in 
operator situation awareness and workload. We found 
that when pilots were responsible for separation, they 
had higher levels of awareness, but not necessarily 
higher levels of workload. When controllers are 
responsible and actively engaged, they showed higher 
workload levels compared to pilots and changes in 
awareness that were dependent on sector 
characteristics. 

Introduction 

The Next Generation Air Transportation System 
(NextGen) is intended to modernize and increase the 
effectiveness of the national air traffic management 
system in the U.S.A. [1]. By 2025, NextGen is 
anticipated to accommodate as much as three times 
(3X) current day air traffic, while increasing its 


efficiency (e.g., shorter flight routes, shorter time on 
ground, fewer delays, etc.) and maintaining safety. 
Many organizations have been working toward 
NextGen goals by researching and developing 
advanced technologies such as controller-pilot 
datalink communications, advanced cockpit displays 
of traffic, weather, terrain, conflict alerting and 
resolution tools, and semi-autonomous automated air 
traffic management agents. New procedures and 
operational concepts for NextGen, such as trajectory- 
oriented operations and performance-based 
navigation procedures, are also being evaluated for 
use [2]. Because the human operator will still be an 
integral part of the air traffic management system, 
NextGen concepts of operation and technologies 
must be evaluated to determine their influence on 
human operator performance, workload, and situation 
awareness. 

Situation awareness refers to an operator’s 
understanding of his or her environment [3]. High 
situation awareness is needed for an operator to 
function optimally in a complex system such as air 
traffic management [3, 4]. Many aviation accidents 
that occur as a result of human error have been 
attributed to low situation awareness [4], Thus, new 
systems and technologies being developed for 
NextGen must also assist operators in maintaining 
high situation awareness. Mental workload refers to 
the relationship between the amount of processing 
capability that an operator has available and the 
demand for those resources required by the task [5]. 
Workload is known to yield a curvilinear relationship 
with operator performance, with performance being 
negatively influenced by extremely low or high levels 
of workload [6]. With regard to separation 
assurance, workload varies as a function of traffic 
density [7], with workload increasing drastically after 



a certain threshold is reached. Thus, new NextGen 
tools and technologies must be evaluated in terms of 
their impact on operator workload. 

This simulation is one of a series of studies in an 
on-going line of research aimed at optimizing 
situation awareness and workload metrics. We used 
online probes to capture operator situation awareness 
and workload. Online probe latency has been shown 
to be more predictive of performance than offline 
probes [8] and related to performance metrics 
assumed to depend on situation awareness [9, 10, 11]. 
Moreover, past simulations we conducted show that 
the online probes can discriminate between levels of 
awareness with different degrees of automation and 
training [12, 13]. Using a pilot conflict resolution 
task with differing levels of automation, [12] found 
that probe latencies were significantly longer in a 
fully automated condition compared with conditions 
in which pilots had to generate resolutions manually 
or were allowed to evaluate and modify suggested 
resolutions. The shorter latencies were indicative of 
higher situation awareness of the information being 
probed through active engagement with the conflict 
resolution task. Vu et al. [13] found longer probe 
latency for student controllers to questions involving 
projection into the future compared to questions 
relating to the present state of events in a simulation 
environment. Since more situation awareness is 
needed to accurately project future states, the probe 
latency appears to be a sensitive metric for capturing 
different degrees of situation awareness. 

We selected three concepts of operation that 
distributed separation responsibility across pilots, 
controllers, and automation. It is worth noting that 
these concepts of operation were chosen because they 
were hypothesized to alter the workload and situation 
awareness of the human operators, allowing us to test 
our situation awareness and workload metrics. We 
do not imply that these concepts are being endorsed 
by any agency for implementation in NextGen. In 
Concept 1, pilots with equipped flight decks were 
responsible for conflict identification and resolution 
between Ownship and other equipped aircraft, aided 
by a conflict detection and resolution 
tool. Controllers managed only conflicts between 
unequipped aircraft. In Concept 2, conflict 
identification and resolution was managed by air 
traffic controllers and automation, with more 
conflicts assigned to the controller than to the 
automation. In Concept 3, conflict identification and 


resolution responsibility was also allocated to a 
human controller and automation system, but more 
types of conflicts were assigned to the automation 
system than to the human controller. 

Methods 

Participants 

Eight experimental pilots were tested during each 
week of a two-week study. However, the 
performance, workload, and situation awareness data 
are only reported for the second week of data 
collection due to equipment failures on some of the 
experimental runs during the first week. Table 1 
shows the number of hours flown for pilots in Week 
2. Five of the pilots were at the professional ra nk of 
captain and three were at the ra nk of first officer. 
None of the participants had any prior experience 
with merging and spacing operations, although three 
of the pilots had experience flying a continuous 
descent arrival (CDA). 


Table 1. Number of Pilots in Each Grouping of 
Total Flight Hours in General and with a “Glass” 
Cockpit 


Total 

hours flown as 
a line-pilot 

N 

Total 

hours flown in 
"glass" cockpit 

N 

1-1000 

1 

1-1000 

4 

1001-3000 

0 

1001-3000 

1 

3001-5000 

4 

3001-5000 

3 

>5000 

3 

>5000 

0 


A total of four experimental controllers were run, 
two in each week of data collection. As with the 
pilots, only data from Week 2 are reported in detail. 
The experimental controllers in Week 2 were retired 
radar certified controllers, with one having 34 years 
and the other 25 years of civilian air traffic control 
experience. 




Simulation Environment 

The simulation environment was produced by the 
Multiple Aircraft Control System (MACS) software 
[14]. The merging and spacing and weather 
avoidance tasks were supported by the Cockpit 
Situational Display (CSD) software [15]. Both 
MACS and the CSD were developed at NASA Ames 
Research Center by the Airspace Operations 
Laboratory (AOL) and the Flight Deck Display 
Research Laboratory (FDDRL), respectively. 
Included in both software programs were tools that 
supported generation of conflict resolutions [16] and 
a version of the auto-resolver tool that could 
automatically generate a resolution upon request 
from the pilot or air traffic controller. The 
algorithms used for the auto-resolver tool were also 
used by a ground-based auto-resolver agent, which 
autonomously uplinked resolutions to the equipped 
flight decks in certain concepts of operation. 

The airspace used in the simulation mimicked 
Kansas City Air Route Traffic Control Center (ZKC) 
and Indianapolis Air Route Traffic Control Center 
(ZID), with controllers managing Sectors ZKC 9- and 
ZID 9 1 . Traffic in each sector was modeled after real 
traffic feeds, but was modified to create a 3X traffic 
density environment. First, the airspace consisted of 
a larger area by combining a high and super-high 
sector. With the additional airspace provided by the 
“super sector” (see Figure 1), additional aircraft was 
added to the current traffic flows to load the airspace 
with 3X current day traffic. Surrounding the two 
experimental sectors that were managed by the 
participant controllers were adjacent sectors that were 
managed by students and staff working in the Center 
for Human Factors in Advanced Aeronautics 
Technologies at California State University Long 
Beach (CSULB). These “ghost” sectors were needed 
to make and receive handoffs from the two 
experimental sectors. 

Aircraft populating the simulation were 
designated as either TFR (Trajectory Flight Rules) or 
IFR (Instrument Flight Rules). TFR aircraft had 
equipped flight decks with conflict detection and 


resolution tools. TFR aircraft were not directly 
managed by the human controller and interacted with 
the controller only under specific circumstances 
(depending on the concept of operation being 
employed and the phase of flight). The IFR aircraft 
consisted of the non-equipped flight decks, and these 
aircraft were always managed by the human 
controller. All experimental participants flew 
simulated desktop stations of TFR aircraft. 


Weather Prior To Sector90in Sector91in 

Active Sectors Simulation Simulation 



Figure 1. Illustration of the Simulation Airspace. 
Weather was Located to the West of Sector 90. 
Sector 90 was a “Super Sector” Consisting of 
ZKC 90 and 14. Sector 91 was a Super Sector 
Consisting of ZID 91,81, and 17. 


Both pilots and controllers were given several 
advanced tools for separation assurance. The first 
tool was the conflict probe, which detects conflicts up 
to 8 minutes prior to Loss of Separation (LOS) and 
alerts the pilot/controller (highlights aircraft in amber 
on the flight deck and flashes red on the controller 
radar; time to LOS was indicated on both displays). 
The second tool was the Route Assessment Tool 
(RAT) for pilots and the Trial Planner for air traffic 
controllers. The RAT and trial planner allows for 
manual creation of flight plan changes. Because this 
tool is coupled with the Conflict Probe, it allows the 
operator to determine whether proposed route 
changes are conflict free. Pilots can also use the 
RAT to re-route around weather. The third was the 
auto-resolver tool, which allows both pilots and 
controllers to request a resolution for a conflict. 
Finally, an auto-resolver agent was provided in 



Concepts 2 and 3. The auto-resolver agent 
functioned autonomously to resolve conflicts 
between designated aircraft (e.g., TFR-TFR or TFR- 
IFR) once delegated this responsibility at the start of 
a scenario. 

A distributed air-ground simulation environment 
was employed. Conducted over the Internet, four 
research labs participated in the simulation. 
Experimental pilots were located at FDDRL at 
NASA Ames Research Center. FDDRL also served 
as the simulation hub, consisting of the simulation 
manager, the voice server, and supporting 
workstations. The participant air traffic controllers 
were located at CHAAT CSULB. CHAAT also 
hosted confederate or “ghost” air traffic control 
sectors and pseudopilot stations which provided 
additional aircraft in the simulation that were not 
managed by the participant pilots being tested. 
Additional pseudopilot stations were located at the 
Systems Engineering Research Laboratory (SERL) at 
California State University Northridge, and the 
Human Integrated Systems Engineering Laboratory 
(HISEL) at Purdue University, to emulate 3X traffic 
density. 

Concepts of Operation 

Three concepts of operation that distributed 
separation responsibility across pilots, controllers, 
and automation were tested (see Table 1 for a 
summary). Again we emphasize that these concepts 
were selected because they were hypothesized to 
influence operator performance, workload, and 
situation awareness, allowing us to evaluate our 
probe tool. 

In Concept 1, TFR pilots had the capability, 
responsibility, and authority for separating their 
ownship from other aircraft using the advanced 
traffic separation tools provided. Pilots made route 
modifications for traffic and weather avoidance and 
executed them; they did not have to datalink route 
modifications to a controller for approval. The air 
traffic controller was not responsible for TFR-TFR or 


TFR-IFR conflicts, except for the arrivals on their 
CDA. During the CDA, TFR pilots were instructed 
not to use the auto-resolver or RAT tools; the 
controller was to monitor them for conflicting traffic. 
TFR pilots were given voice frequencies to monitor 
but were told that they would not receive clearances 
from the air traffic controller unless they have 
discontinued spacing and given control back to the 
air traffic controller. IFR pilots did not have 
equipped flight decks and were always under the 
control of the human controller. IFR pilots received 
clearances via datalink, but they also monitored a 
voice channel. Controllers issued voice commands to 
any aircraft when needed. The controllers were 
responsible for resolving IFR-IFR conflicts. For 
TFR-IFR conflicts, the controllers were told that the 
pilot of the TFR aircraft was responsible for 
resolving the conflict. 

In Concept 2, TFR pilots had the capability to 
separate Ownship from other aircraft by using the 
advanced traffic separation tools provided. However, 
TFR pilots did not have responsibility or authority for 
traffic separation. If the TFR pilots decided to use 
the advanced tools to generate a conflict resolution, 
they had to datalink the proposed solution to the air 
traffic controller for approval. Once the proposed 
route modification was received, the controller could 
approve/disapprove the request or issue another 
clearance. As with Concept 1, TFR pilots were given 
voice frequencies to monitor. IFR pilots were under 
the control of the human controller as in Concept 1. 
The human controller was responsible for resolving 
IFR-IFR and IFR-TFR conflicts. For IFR-TFR 
conflicts, the controller was to move the IFR aircraft. 
Conflicts between TFR-TRF aircraft were the 
responsibility of the autoresolver agent. 

In Concept 3, pilots did not have conflict 
detection or resolution tools. Pilots did have the 
RAT for weather avoidance and for making other 
routing requests, but the RAT was not coupled with a 
conflict probe. All requests from pilots had to be 
datalinked to a controller for approval. All pilots 
monitored a voice frequency but most of the 



communication between pilots and controllers 
occurred via datalink. The human controller was 
responsible for resolving IFR-IFR conflicts. IFR- 
TFR and TFR-TFR conflicts were delegated to the 
auto-resolver agent. For IFR-TFR conflicts, the TFR 
aircraft was burdened to move unless it was an arrival 
aircraft on the CDA. 


Table 1. Tools Available and Operator 
Responsibility for Each of the Three Concepts of 
Operations 


Concept 1: 

Pilot Primary; 

Controller 

Secondary 

Concept 2: 

Controller 
Primary; Auto- 
resolver Agent 
Secondary 

Concept 3: 

Auto-resolver 
Agent Primary. 
Controller 
Secondary 

• Pilots have 
conflict 
detection and 
resolution 
tools and are 
responsible 
for solving 
conflicts with 
Ownship 
(75% of total 
conflicts) 

• ATCs are 
responsible 
for resolving 
25% of total 
conflicts 

• Auto- 
resolver 
agent is not 
responsible 
for solving 
any conflicts 

• Pilots have 
conflict 
detection and 
resolution 
tools but are 
not 

responsible 
for solving 
any conflicts 

• ATCs are 
responsible 
for resolving 
75% of total 
conflicts 

• Auto- 
resolver 
agent is 
responsible 
for resolving 
25% of total 
conflicts 

• Pilots do not 
have conflict 
detection and 
resolution, 
and are not 
responsible 
for solving 
any conflicts. 

• ATCs are 
responsible 
for resolving 
25% of total 
conflicts 

• Auto- 
resolver 
agent is 
responsible 
for resolving 
75% of total 
conflicts 


Tasks and Procedures 

The pilots and controllers interacted in a real- 
time simulation environment. There were twelve, 90- 
minute scenarios, with four replications of each 
concept of operation. Separation requirements for all 
aircraft were 5nm lateral and 1000 feet vertical. In 
addition to performing the tasks specific to each 
operator role, described below, all participants were 
asked to answer situation awareness and workload 
probes. 

Within the first 10 minutes of the scenario, pilots 
were assigned a lead aircraft and given spacing 
instructions. Pilots were asked to fly the Sea Biscuit 
One arrival into Louisville Standiford International 
Airport (SDF) while maintaining separation from 
other traffic (in Concept 1 only), avoiding convective 
weather, maintaining the assigned spacing interval 
relative to a lead aircraft at the final approach fix, and 
complying with Sea Biscuit One’s altitude and speed 
restrictions. At the Sea Biscuit One fix, experimental 
aircraft performed a CDA to the 17-R runway. 
Experimental pilots were to notify air traffic control 
when they had discontinued spacing; at that time the 
aircraft was under the control of the air traffic 
controller. Depending on the condition being run, 
pilots were given a static depiction of weather on the 
CSD in the form of a Nexrad 2D or 3D display. 

Controllers were asked to manage traffic in their 
sector. IFR traffic was displayed at full brightness 
and TFR aircraft were dimmed unless they were in 
conflict with an IFR aircraft. Controllers had a static 
image of weather (always located to the west of ZKC 
90) on their radar that was similar to the Nexrad 2D 
display. Maintaining separation of aircraft to which 
they have been assigned responsibility was the task 
of greatest priority. In addition, controllers were 
asked to acknowledge voice check-ins for IFR 
aircraft, give clearances via pilot-controller datalink 
communications, re-sequence arrival aircraft on 
request, and provide traffic advisories, when time 
permits. The datalink clearances reflect trajectory 
changes to the aircraft flight plans and were to be the 
preferred mode of communication between the 




controller and pilot; however, to maintain safety, all 
pilots were monitoring the controller’s voice 
frequency and could respond to verbal clearances 
when necessary. To alleviate controller workload, all 
handoffs and frequency changes were automated. 

Situation Awareness and Workload Probes 

Starting at 4 minutes into the scenario, situation 
awareness and workload probes were presented at 3- 
minute intervals. The online probes were 
administered following the Situation Present 
Assessment Method sequence of events 
[8]. Operators were given an audio alert and a visual 
ready prompt to indicate that a probe question was 
available. When the operator indicated being “ready” 
by pressing the “ready” prompt on a touch screen, a 
probe question was administered. The question 
either queried the operator about workload [Rate your 
workload on a scale of 1 (very low) to 5 (very high)] 
or situation awareness. Sample situation awareness 
questions for pilots are listed below: 

• “Will Ownship overtake UPS419?” 

• “What was the last command you issued?” 

• “How far will you deviate laterally for 
weather?” 

Sample situation awareness questions for pilots 
are listed below: 

• “Which quadrant of your sector currently has 
the most eastbound traffic?” 

• “Will UPS914 and AAL114 maintain lateral 
separation if no further action is taken?” 

• “What was the last command you issued via 
datalink?” 

These questions were designed to query for 
information about traffic flows, conflicts, merging 
and spacing status, and weather. Some questions 
were general and could be asked at any place in the 
scenario (e.g., what was the last command you 
issued?) and to any operator (pilot or controller). 
Other questions were tailored to the scenario for 
relevance (e.g., “How far will you deviate laterally 
for weather?”) was asked to a specific operator, the 
pilot, prior to the aircraft passing weather. 


Results and Discussion 

Pilot workload and situation awareness data from 
the simulation have been reported in detail by in 
other papers [17, 18], The present paper compares 
pilot workload and situation awareness to 
controllers. Because there were only 2 controllers, 
and each controller was responsible for a different 
sector, inferential statistical analyses were not 
performed. Rather descriptive data are reported and 
trends between controllers and pilots are discussed. 

Performance 

Loss of Separation (LOS) was the performance 
metric of interest for separation assurance. The 
number of LOS was determined for pilots and 
controllers as a function of the concept of operation 
being employed. The data from 1 pilot had to be 
removed from the sample due to non-compliance 
with experimental instructions. On average, 
experimental aircraft had one LOS on every other 
run. The number of LOS was higher in Concept 1, 
when the pilots were responsible for resolving the 
conflicts, than in Concepts 2 and 3, when the 
controller and auto-resolver agent were responsible 
(see Figure 2). It is important to note that the traffic 
density was 3X and that the responsibility for 
resolving conflicts for pilots is one that they are not 
given in current day operations. 
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Figure 2. Mean Number of LOS per Experimental 
Flight Deck and Scenario as a Function of 
Concept of Operation. 



For the two participant controllers, the number of 
LOS for each sector as a function of concept of 
operation is shown in Figure 3. On average, the 
controller for sector 90 had 3 LOS on each run, and 
the controller for sector 91 had 1 LOS on each run. 
Again, it is important to note that the traffic density 
was 3X in these scenarios. Also, it is worth noting 
that the sector characteristics were different for sector 
90 and 9 1 . Sector 90 was a larger sector with traffic 
flows being affected by convective weather, west of 
the sector. It also contained the Top of Descent 
(TOD) for arrival aircraft into SDF. The controller 
for sector 90 was also responsible for re-sequencing 
of the experimental aircraft if requested by the pilot. 
Sector 91 contained the merge point for the 
experimental aircraft; the controller for that sector 
managed crossing traffic so that it did not interrupt 
the arrival streams. The sector 91 controller was also 
given the responsibility for issuing clearances to the 
arrival aircraft to maintain the spacing interval if the 
aircraft discontinued spacing. Given the varied 
nature of the two sectors, it is likely that differences 
between the numbers of LOS in each reflected sector 
properties rather than differences in controller ability. 
One pattern that did emerge for both controllers was 
that the number of LOS was greater in Concept 2, 
when they were responsible for 75% of conflicts in 
the sector than in Concepts 1 and 3, when they were 
responsible for only 25% of the conflicts. Note also 
that the mean number of LOS were slightly higher in 
Concept 1 (shared responsibility with pilots) than 
Concept 3 (shared responsibility with an automated 
agent). 
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Workload 

The probe technique allowed for two different 
measurements of workload. The first is the ready 
latency, which is the time from presentation of the 
ready prompt to the controller indicating that s/he 
was ready for a probe question by pressing the 
prompt on a touch screen display. If workload was 
low, the operator should have been able to accept the 
ready prompt more quickly than if workload was 
high. The ready prompt timed-out after one minute. 
For the workload probes, a second metric is obtained, 
the workload rating (1-5 scale). For the workload 
probes, there was a significant correlation between 
the ready latency and workload rating, r = .25, p < 
.001 [17]. Thus, the ready latency was used as a 
measure of workload. 

Ready latencies were influenced more by time in 
scenario than by concept of operation. Pilots had 
higher ready latencies during the first 9 minutes of 
the scenario, when they encountered the weather, and 
after 45 minutes into the scenario, when they were 
landing. Controllers showed higher latencies after 27 
minutes into the scenario, when the experimental 
aircraft started to enter their sectors. 

For the workload ratings on the workload probes, 
pilots reported lower levels of workload (Ms = 1.7 to 
1.8) compared with controllers (Ms = 2.5 to 3.0) 
across all concepts. Controllers reported that 
workload was highest in Concept 2 (M = 3.9), where 
controllers had the most responsibility. Lower 
workload levels were reported for Concept 1 (M = 
3.1, when pilots were responsible for the majority of 
conflicts) and Concept 3 (M = 2.5, when automation 
was responsible for the majority of conflicts). 

Situation Awareness 

The probe technique allowed for two different 
measurements of situation awareness. First, the 
percentage of probes answered correctly could be 
used as an indicator of awareness. Lower errors 
indicate higher awareness. The second metric is the 
probe latency. It is assumed that the lower the 
latency, the higher the awareness of the operator for 
the information being queried. 


Figure 3. Mean Number of LOS per Experimental 
Sector and Scenario under 3X Traffic as a 
Function of Concept of Operation. 
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Figure 4. Top: Mean Percent Error to Situation 
Awareness Probes for Pilots and Controllers as a 
Function of Concept of Operation; Botton: Mean 
Situation Awareness Probe Latency for Pilots and 
Controllers as a Function of Concept of Operation 

As shown in Figure 4, pilots showed fewer errors 
and shorter latencies to the probes in Concept 1, 
when they were responsible, than in Concepts 2 and 3 
(detailed analyses can be found in [18, 19]). For 
controllers, the data pattern was less clear. Error 
rates on the probe questions tended to be higher in 
Concept 1 than in Concepts 2 and 3. Flowever, 
because the data were from only two controllers, and 
each one was responsible for a different sector, the 
differences among controller awareness cannot be 
attributed solely to concept of operation. 


Post-Simulation Questionnaire and 
Debriefing 

As noted in the method section, the simulation 
was run over a two week period, with different 
experimental pilots and controllers in each week. 
Due to equipment failure in Week 1, not all data runs 
were collected to allow for appropriate 
counterbalancing of conditions. Although the data 
reported for performance, workload, and situation 
awareness probes were only reported for Week 2 of 
data collection, all 16 pilots and 4 controllers 
experienced all three concepts of operations and were 
given the same post-simulation questionnaire. We 
report the data from all the participants here for a 
more representative picture of operators’ perceptions 
of the three concepts of operations tested. 

Tables 3 and 4 lists the pilot and controllers 
responses to three questions relating to the 
plausibility of the concepts of operation tested, the 
operator perceived workload given the tools 
provided, and the level of operator awareness for 
experimental aircraft in the simulation. 


Table 3. Pilot Responses in the Post Simulation 
Questionnaire (Scale of 1: strongly disagree to 5: 
Strongly agree) 


Question 

Concept 

1 

Concept 

2 

Concept 

3 

Concept (1,2,3) is in 
principle a workable 
concept 

4.53 

4.65 

4.71 

Overall workload is 
manageable using the 
tools provided 

4.41 

4.59 

4.65 

I felt that ATC was 
adequately aware of 
what was going on with 
my aircraft 

2.94 

3.14 

3.59 


In general, all operators indicated that the three 
concepts were workable in theory. Pilots strongly 
agreed that their workload was manageable given the 
tools provided. Controllers indicated their workload 
was the most manageable in Concept 3, where 
separation responsibility was shared between the 
controller and automation, but the auto-resolver agent 
was allocated the responsibility for resolving the 
majority of conflicts. Given that the experimental 
aircraft were all TFR aircraft for which the 


controllers were not assigned responsibility, the 
controllers indicated that they were not adequately 
aware of what the experimental pilots were doing. 
Similarly, the experimental pilots indicated that they 
did not feel that controllers were adequately aware of 
their aircraft. 


Table 4. Controller Responses in the Post 
Simulation Questionnaire (Scale of 1: strongly 
disagree to 5: Strongly agree) 


Question 

Concept 

1 

Concept 

2 

Concept 

3 

Concept (1,2,3) is in 
principle a workable 
concept 

4.25 

4.5 

5.0 

Overall workload is 
manageable using the 
tools provided 

3.5 

3.75 

4.5 

I felt adequately aware 
of what the Exp Pilots 
were doing 

2.5 

2.5 

2.5 


From debriefing, both pilots and controllers 
commented on the level of traffic being high. Pilots 
were less influenced by the traffic level because they 
were mainly responsible for separating their aircraft 
from others. However, controllers had to be aware of 
all aircraft in their sector, so traffic level was a major 
factor influencing their performance. Controllers 
also indicated that they did not mind sharing 
separation responsibility with other operators because 
workload would be reduced if everything worked as 
planned. However, the controllers indicated when 
they were not responsible for a subset of aircraft, it 
was very hard for them to be able to assist those 
aircraft subsequently. For example, when the auto- 
resolver agent was unable to resolve a conflict in a 
timely manner, the pilot would call the human 
controller for assistance. In these circumstances, the 
controllers sometimes could not locate the aircraft in 
a timely manner to assist in the conflict resolution. 

Conclusions 

Pilots and Controllers were affected differently 
by the three concepts of operation. Pilots indicated 
that all concepts were workable and showed little 
change in workload across the three concepts. 
Moreover, pilots showed higher levels of awareness 
when they were actively engaged in separation 


assurance responsibilities in Concept 1 . Although the 
controllers also indicated that all concepts were 
workable, they showed sector-specific changes in 
awareness and workload depending on concept of 
operation. Pilots tended to revert back to controller 
intervention when for assistance when their solutions 
(or the auto-resolver agent’s solutions) failed. In 
these situations, though, the human controllers 
showed very little awareness of the experimental 
aircraft and were not be able to help the pilots with 
resolving conflicts. This finding, although 
preliminary, has implications for models of NextGen 
that put the human controller in a back-up role for 
other human operators or for automation failures. 
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